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SUMMARY 



In this paper, w present a model for an Integrated system used for 
job-aiding, training, and performance assessment. The model Is driven by 
updatable job aids, by Integrated man-machine heuristics, and by an 
expanding matrix of maintenance activities. Responsible to AFHRt 
Initiatives, as we understand them, and compatible with pending and probable 
system innovations, the model Is designed to serve well Into the 21st 
Century. 

The model uses the job-aiding system as the base which Is kept 
up-to-date by computer networked storage and retrieval. In our model, this 
system Is part of an "expert systan"; that Is, a system which can "learn." 
All Inputs and outputs are In natural, human language presented In a 
user-friendly series of displays and menus. 

The model also provides for training and performance assessment. To 
create training modules, the computer subsystem Implements the appropriate 
job aid by presenting It In a training frame; to create a performance 
assessment battery, the computer subsystem presents the job aid after 
filtering It through a linguistic transformation which turns It into a 
case study or, if appropriate, a series of questions. 
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PREFACE 



During the Sunroer of 1984, under the sponsorship of the Air Force 
Human Resources Laboratory* ^ ^rkmi at the Training Systems Division at 
Lowry Air Force Base. Colorado. Vfe attempted to create a concretely focused 
model, ^sed on current knowledge about computer-asslsted Instruction (CAI), 
about artlflcal Intelligence (AI). about language and thought. This 
document and the briefing we gave present the gist of our idea. 

Our work was informed by the efforts and support of Or. Joseph 
Yasutake, Maj. Dale Baxter. Maj. Richard Bolz, Dr. Roger Pennell, Mr. Brian 
Call man and others too numerous to mention. Especial acknowledgement of 
Maj. Hugh Burns, our host, is conveyed both ^ this notice and by the docu- 
ment itself— he was the prime force for our coming together; he is a ready 
li'-tener, an astute crltit, and a scholar. 

The opinions advanced within this report do not necessarily reflect 
those of the Air Force. Air Force Systems Command, or the Air Force Human 
Resources Laboratory (Training Systems Division); they are ours alone. 
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I. INTRODUCTION 



As newer systems of all types are brought Into use In the Air Force, 
the job of fialntainlng them becomes more and more complex. This Increased 
coin)l8x1ty Is due both to the Increased complexity of the equipment Itself 
and to the necessarily enlarged uita base. (New systems do not usually 
Instantly supplant their predecessor systems.) 

Maintenance Involves a four-fold planning effort: assessing who needs 
to learn to do what, teaching that job skill, aiding with an up-to-date 
and well-focused system, and providing the equipment, material, and 
supervision necessary to allow maintenance personnel to utilize their 
skills. The first three are training Issues as shown In Figure 1. The 
fourth is an organizational support Issue. 

In this tech nical paper, we present a rationale and a concept for 
dealing with the three training Issues; we also survey their impact on those 
organizational support Issues of which we are aware; and we provide examples 
of now such an integrated system might work. 

Crucial to the proposed solution Is the use of job aids, computer 
networked storage and retrieval, and constantly updating and learning 
systems (so-called "expert" systems). All inputs and outputs (I/Os) of the 
proposed system are based on use of natural, human language (could be any 
specific language). We propose a continually updating and learning system 
with the ability to deal with unprogrammed problem-solutions. 

Nothing we have proposed in this document, we believe, relies on 
technology not in existence today. Indeed, the strength of this model lies 
in large part in reliance upon state-of-the-art capabilities combined with 
a unique design for packaging, driving, and using the system. 




FIGURE 1 SYSTEM MAINTENANCE PLANNING 
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The redundancy of training and job aiding systems In the Air Force, 
while certainly understandable, seems to us wasteful and lamentable. One 
goal of this effort was to eliminate that redundancy by using job aids as 
the stockpile from which training materials are taken. Training exercises 
are therefore completely up to date, because the design provides for a 
continuously updating job-aiding system. These job aids and training 
exercises form the basis for the non-sociometric aspects of performance 
assessment. Thus, it Is an integrated model. The organizational support 
requirements are largely data-based; we include a discussion of how that 
support mechanism might trork. 

It needs to be stressed that we did not design the actual system. In 
fact, in our short consulting assignment, we did not study all possible 
applications of job aids, or of training systems, or of performance 
assessment. This is a concept paper. 

What we have done is work (»it a solution to this problem: Create the 
concept for an interactive, computationally and natural language driven, 
"expert system which will improve the delivery of training. 

We extended the problem to include job aiding and performance 
assessment and narrowed it to currently available technology (with, to be 
sure, an eye toward the innovations foreseen for 1985 to 2025). 

Our report is presented in three major i^ubsections: 

II. The Integrating Model 

III. Updating the System 

IV. Anticipating Clearly the Future 
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II. THE INTEGRATED MODEL 

The heart of the proposed design Is the job aid data base which Is 
called on and augmented for training purposes. The job aid data base 
becomes also the base upon which performance measurement toels are built. 
In this section we present the concept of the Integrating/Integrated model; 
the presentation Is divided Into three sections: 

A. Job-Aiding 

B. Training 

C. Performance Assessment 
A. Job Aiding 

Classic Expert Systems 

One of the uses of artificial Intelligence (AI) has been to develop 
expert systems. An expert system Is the formalization of the practices 
(both conscious and Intuitive) of experts. These formalizations can then be 
reduced to step-by-step algorithmic procedures for the ^Idance of novices, 
who, with the assistance of the procedures, can then mimic the performance 
of experts. 

Such an approach has certain strengths and weaknesses, as Figure 2 
Illustrates. Figure 2 shows the Interaction of two different dimensions: 
the skill level of the user of the expert system and the difficulty of the 
task to be performed. In Cell 1, we see the most powerful use of expert 
system procedures. The user has a low skill level and thus needs the 
maximum amount of guidance. The task Is highly routine and thus well 
described and documented by the procedures. As the user gains experience 
with routine problems, his/her dependence on step-by-step procedure lessens. 
In Cell 2, we see that a highly skilled individual no longer needs the 
system at all, or more accurately, his/her requirement for the system 
changes from needing a guide book to needing a reference book. Thus, an 
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FIGURE 2 MODEL OF EXPERT SYSTEMS 



^ 14 



expert system that has a skip-ahead or random access format could serve 
the needs of the highly skilled Individual whereas a highly linear expert 
system which requires lock-step use would not. 

The other dimension In Figure 2 reflects the "fit" of the expert 
system to the problem or task with which the Individual is working. 
Presumably, the expert system Is highly effective in stealing with routine 
problems. However, not all problems are routine and there will be 
situations In which the expert system does not "fit" the problem very well. 
In Cell 3, a novice user cannot use the expert system because the individual 
lacks the skills to adapt or adjust the system to fit the unique problem. 
In Cell 4, however, we see that a highly skilled Individual Is able to 
deviate from or make adjustments to the expert system to cope with the 
Idiosyncratic nature of the problem at hand. As In Cell 2, the highly 
skilled Individual would be assisted by an expert system that allows for 
flexible use. 

The approach we have described above might be called "the cookbook 
approach" to expert systems. A novice cook will follow the procedures 
rather slavishly (Cell 1 In the chart above). As the cook makes the same 
Item over and over, she/he becomes less and less dependent on the recipe, 
until a point (Cell 2) Is reached, at which point the cook merely refers to 
the recipe to check on a specific Ingredient or step In the process. When 
attempting to prepare an exotic dish for which no reliable recipe exists, 
the novice Is In Cell 3 — he is unable to adapt existing recipes to meet 
his needs, whereas a highly skilled chef can combine existing recipes in 
an innovative way to prepare the dish (Cell 4). 

One of the functions of an expert system Is to replicate the expert's 
ability to solve problems. The expert system's approach to problem solving 
is usually represented as a branching diagram. The top node (or state) in 
the branching diagram represents a statement of the problem. Branching from 
this node are two or more nodes which make mutually exclusive statements 
about the higher node, seen in Figure 3. 
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A 

(CAR WONT START) 


1 1 
B C 


D 


(STARTER WON'T (STARTER TURNS 
TURN OVER) OVER. BUT ENGINE 

WON'T CATCH) 


(ENGINE STARTS. 
BUT 

IMMEDIATELY DIES) 


FIGURE 3 BRANCHING DIAGRAM 
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Subsequent branches will then describe each of the three states (nodes 
B, C, D) with more detailed statements until an exhaustive set of terminal 
states, or solutions, are reached. For example, in Figure 3, one terminal 
state for node B (starter won't turn over) will be the determination that 
the battery is dead. 

In abstract terms, the expert system approach to problem solving may be 
characterized in the following manner: 

1. The expert's knowledge is captured in a branding diagram that has 
a single initial state (the statement of the problem), a finite number of 
intermediate states (analyses of the problem), and a finite number of 
terminal states (solutions to the problem). 

2. Branching from the initial state (and from all subsequent states, 
which are starting points for their dependent branches) are a finite number 
of mutually exclusive and exhaustive statements about the higher state (or 
node), only one of which in any given situation can be true. 

3. The branching diagram has the technical characteristics of a 
context-free, phrase-structure grammar (e.g., all intermediate nodes nwst 
have branches and each of these branches must end in a terminal state, i.e., 
a solution; branches must never cross over each other). These 
characteristics guarantee that from any terminal state there Is only one 
possible pathway back up to the initial state and only one possible pathway 
from the Initial state to a given terminal state (Chomsky, 1963). 

Suppose that for some reason the intermediate states (nodes) in Figure 
3 ./ere not available to the user of the expert system. All that the user 
has is the Initial state (the problem) and the terminal states (the 
solution). Could the user still solve the problem? The answer, of course, 
is "yes," but only in a very inefficient way. The user could find the 
solution by merely testing sach of the terminal states one by one until he 
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or she found the one that was true. What we have just described Is a trial 
and error system, a system that dees not depend on analyzing the problem. 
Users have no basis to make a connection between the problem and the 
solution because they lack the Intermediate states which are the analyses of 
the problem. 

The expert system we have described rests on two k^ assuinptlons: (1) 
The branches from each node are. In fact, mutually exclusive and exhaustive. 
For exanple. In Figure 3, if B is a true statement about A, then C and 0 
cannot also be true statements. Moreover, there Is no possibility that 
there Is a fourth statement about A, a new node E, which Is also true. Put 
in different terms, the expert system requires that the sum of the 
probabilities for all the branches from each node total 100% (I.e., unity). 
(2) The user of the expert system (the branching diagram) always has 
sufficient Information available to correctly determine which branch Is a 
true statement for the particular problem. 

In the real world, neither of these assumptions Is valid- Branches do 
not have to be mutually exclusive — there can be more than one true 
statement about a node. For example. In the sample problem. It could be the 
case that In addition to the battery being dead, there is something wrong 
with the starting motor. In the real world. It is seldom the case that the 
sum of all the probabilities for describing a given state Is IW. There is 
always the chance of discovering a new possibility that the expert system 
had not anticipated. Finally, we live in an uncertain world. Ho test 
equipment is 100% reliable under all conditions. Some symptoms of a problem 
are intermittent: Ihey appear and disappear. The evidence that one draws 
upon to make a decision between alternative explanations 1s often 
inconsistent and even contradictory. 

In the real world, a problem does not have to have a single cause. 
There can be multiple failures In the same system, and failures can interact 
with one another with consequences that are quite .-ewildering to the 
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observer. The problem and the symptoms of the problem can be disassociated; 
for example, the failure of a component In system A may be apparent only In 
a malfunction of a component in ^stem B. In other words, it Is not always 
obvious In which branching tree diagram the problan resides. There is not 
necessarily a one-to-one correspondence between a problem and its symptoms. 
For example, the same problem in different circumstances may exhibit 
different symptoms; conversely, two different problems may have identical 
symptoms. 

The result of this real world complexity and uncertainty is to 
effectively remove the Intermediate nodes from the expert system. In short, 
real world complexity and uncertainty reduce the expert system problem 
solving tree to, at best, a trial and error systen. 

Interactive Expert Systems 

The power of the expert system model is that It guarantees the solution 
to a problem. The limitation of the model is that it requires a set of 
inputs (a guarantee that the branches from each node are nwtually exclusive 
and exhaustive and that the user alw^s has sufficient information to 
correctly choose among the branches) that cannot be achieved in the real 
world in any but the most simplistic circumstances. 

We propose a new model of expert system that accepts the limitations 
noted above. This system does not presuppose that the branches from eac^^ 
node are mutually exclusive and exhaustive, nor does it not presuppose that 
the user always has sufficient information to correctly choose among the 
branches. This model assumes that the normal state of affairs In the real 
world is uncertainty. The user can never assume that the expert system 
knows all the answers (and even if it did, the user can never assume that 
he/she has all the information required to find that right answer). 
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The proposed model Is much more realistic In its assumptions about the 
real world than is the conventional model of expert systeire. The model 
gains in practicality, but loses in power: It cannot guarantee a correct 
solution to a problem. It provides the user with a set of inputs which the 
user employs in an interactive manner to attack the problem. It is the user 
who solves the problem, not the system. The user interacts with the expert 
system, using the expert system to provide him/her with a process for 
identifying alternative solutions to the particular problem and a rich body 
of data which enables him/her to make intelligent guesses about the most 
likely solution. For this reason, we call the model an "interactive expert 
system." 

The interactive expert system assists the user In moving by stages of 
successive approximation from initial recognition and definition of the 
problem to discovery of the solution. The system provides an analytical 
procedure and a data set which the user can draw upon to find which 
alternative among many has the best fit with the actual problem. The user 
must then verify that the best ^ess is indeed correct. If it is not, the 
user must then choose the next best guess. 

To see how an Interactive expert system approximates what experts 
actually do in solving problems under conditions of uncertainty, we will 
examine how a physician diagnoses an illness. Let us. Imagine that a patient 
comes to a doctor complaining about fatigue and general malaise. Before 
attempting any specific diagnosis, the doctor draws upon two additional 
pieces of information: (1) a determination of the patient's general state 
of health at the present moment, by means of a simple, standardized set of 
tests (the patient's vital signs): height, weight, pulse rate, body 
temperature, and blood pressure; and (2) a determination of the patient's 
medical history by means of a standardized self -reporting form. At this 
point, the doctor has three pieces of Information: I) the patient's initial 
description of syn^toms, 2) the patient's current state of health (the vital 
signs), and 3) the patient's medical history. We will collectively call 
these three pieces of information the patient's initial symptom set (ISS). 



The first step in the physician's diagnostic procedure is to match the 
ISS against the symptom set of diseases and ailments of which the doctor Is 
aware. A critical component of the doctor's expertise is his/her knowledge 
of diseases and ailments, together with the symptoms that are associated 
with each of them. In more foniial terms, the doctor does a matching sort, 
selecting for furthe consideration those diseases and ailments that havo 
symptoms conpatlble with the ISS and discarding from consideration those 
that do not match the ISS. This matching sort accomplishes two things: 1) 
It reduces the number of potential solutions to the diagnostic problem to a 
tractable number, and 2) the doctor can draw on knowledge of these potential 
diseases and ailments to make predictions about additional syiH)toms that the 
patient may have beyond the ones already Identified in the ISS. For 
exain>le. Imagine a patient with three salient symptoms (A, B, C) — the 
ISS — and six diseases and ailments that also have these same three 
syin)toms. We may represent this knowledge as seen In Figure 4. The doctor 
can now begin to further narrow the list of potential diseases and ailments 
by seeing if the patient exhibits the additional symptoms D, E, F, and G. 

The ability to create the list of potential diseases and ailments which 
match the ISS Is a critical step in the diagnostic process. Without this 
11st and the additional symptoms associated with the diseases and ailments 
on the list, the doctor has no systematic basis for further diagnosis except 
by trial and error. The additional symptom are generated by this list. 
Without the list, the doctor would have no further symptons to Investigate. 
In other words, additional symptoms do not exist until the doctor knows to 
look for them — a fact especially true of syn^toms that are not physically 
apparent to the doctor, for example, previous events in the patient's 
medical history that were not elicited on the standard medical history form. 

This point is nicely illustrated by an anecdote in one of James 
Herri ot*s stories. Herri ot was treating a dog with the ISS of listlessness, 
loss of appetite, vomiting, and diarrhea. Despite a variety of treatments. 
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the dog was getting worse and worse. One day when Herrlot was examining the 
dog, the dog vomited peculiarly forcefully, which Herrlot immediately 
recognized as a key syn^tom of an ailment that he had not previously 
considered. When Herrlot asked the dog's owners why th^ had not told him 
how the dog had been vomiting, they replied that he had never asked them. 

The physician's ability to create the list of potential diseases and 
allmenv^s is highly significant In two other ways: 1) the doctor has a 
general sense of the relative probability of occurrence of the diseases and 
ailments on the list. Some may be coninon; others, quite rare. All other 
things being equal, the doctor will make a best ^ess that the more common 
Items are the cause of the problem. 2) The doctor knows or can easily find 
laboratory tests which can clinically confirm the existence of the items 
on the list. All other things being equal, the doctor would first employ 
the tests that would confirm (or disconflrm) the presence of the higher 
frequency /most likely items on the list. We may represent the doctor's 
knowledge of each disease and ailment on the list as shown in Figure b. 

Associated with each disease or ailment is (1) a full set of symptoms, 
(2) a probability of occurrence as correlated with other variables such as 
age and sex of the patient, (3) a set of laboratory tests for confirming or 
disconflrming the presence of the disease or ailment, and finally (4) a set 
of treatments appropriate for this disease or ailment. All things being 
equal, the next step In the diagnostic process would be to use the clinical 
tests to confirm the presence of the disease or ailment that has the highest 
probability of occurrence. 

However, all things are not equal. Both the suspected disease or 
ailment and the laboratory tests to verify best guesses are substantially 
affected by other variables — variables of practical importance to the 
doctor. There are two variables that affect the probability of a disease or 
ailment. The first Is practicality. Suppose that the diagnosis has been 
narrowed to a choice between two possible ailments. Suppose further that 
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the treatment of the two ailments is Identical. In the real world, it nay 
be a natter of only academic interest which of the two ailments the patient 
Is actually suffering from; the doctor has no practical reason to continue 
the diagnosis further. For example, the doctor might narrow the list of 
possible diseases to a number of different bacterial Infections. If the 
treatment for all of these different types of bacterirJ Infections is a 
broad spectrum antibiotic. It Is unnecessary to identify which one it 
actually is. 

The second variable affecting the choice of a disease or ailment is 
what might be termed the worst case situation. Some diseases and ailments 
are mich worse than others. If a life-threatening disease matched the 
patient's symptom set, the doctor might well override all considerations of 
probability in order to reassure himself that the patient was not Infected 
with this particular disease. 

There are also at least two important variables In select. ng a 
laboratory test. One is the cost of the test. There are many kinds of 
costs — cost in time, cost in money, cost In discomfort to the patient. 
Some tests are harmful and some even dangerous. Another Is the reliability 
of the test itself. In addition to the possibility of laboratory error, 
some tests are inherently far from being 100% reliable. 

The physician mus*. weigh many variables In test selection. There is no 
algorithm to tell him the best alternative. What Is best in one situation 
may be a poor choice in another situation. For example, the level of risk 
of a particular test may be acceptable with a young patient but unacceptable 
with an elderly patient; the doctor may be reluctant to employ an expensive 
battery of tests for a patient not covered by a health plan; it might be of 
no practical consequence to even diagnose the ailments of an elderly patient 
with a defective Immunological system. 
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The physician's analysis of a patient's Illness denionstrates hew a 
con|)lex diagnostic procedure works. The key idea of the interactive expert 
system model is a matrix that associates the possible causes of the problem 
with (1) the characteristic symptoms of e^ch possible cause, (2) the 
probability of the occurrence of each cause relative to other causes, (3) 
tests by which the existence of each cause can be verified, and (4) a set of 
treatments which remedy or correct each cause. Such a matrix, a "knowledge 
matrix," is depicted in Figure 6. 

This matrix reflects the collective knowledge of experts In the field- 
One of the great advantages of organizing Information in a matrix format is 
that each cell can be Independently updated to reflect new findings or 
conditions. Another major advantage of the matrix is that the user can 
treat each column as an independent variable. As in the case of the 
doctor's diagnostic process, making a best guess about which is the most 
likely cause In a particular situation is not simply a matter of 
probability; the best guess must take into account many practical matters of 
testing and treatment. 

An Interactive expert system, so called because it requires the 
interaction of the knowledge matrix with a user, requires that the user do 
the following in order for the model to succeed: 

1. The user must be able to construct some kind of appropriate ISS for 
the system under analysis. All ISSs will have the same three basic 
elements: (a) a set of specific symptoms that indicate that there is a 
problem (if there were not symptoms of this type then one would never know 
that a problem existed), (b) some Information about the current state of the 
system. I.e., something that corresponds to the vital signs in medical 
diagnosis, and (c) some information about the past history of the system. 
In any particular universe (for example, medical diagnosis or 
troubleshooting a weapons system), there needs to be son© standard format 
for organizing and presenting the ISS. 
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2. The user must match the ISS against the Symptom Set (column 1) 

of the knoviledge matrix. In a large and con^lex system with many possible 
problem causes, this could be done by a cMV"ter search for those causes 
whose symptom set Includes the symptoms In the ISS. 

3. The user must further narrow the list of possible causes by looking 
for previously unnoted symptoms In the system under analysis on the basis of 
the additional symptoms In the Symptom Set of the knowledge matrix. 

4. The user must now make the best guess among the remaining possible 
causes. While the best guess Is strongly Influenced by the Probability Set 
(column 2), the user must also take Into consideration the practicalities 
involved In the Test Set (column 3) and the Treatment Set (column 4). 

5. The user must verify whether the best ^ess was. In fact, correct. 
The user may employ one or more tests from the Test Set, or If the treatment 
Is Inexpensive and simple, the user might skip over the testing state 
entirely and see If the treatment solves the problem. If the best guess was 
not correct, the user must go back to step 4 and make a new best guess, and 
so on until either (a) one of the best guesses about the cause of the 
problem provides the solution, or (b) all of the causes have been elimi- 
nated. 

In this latter case, the user enters Into an entirely different 
diagnostic procedure which we have labeled the "detective mode." The flow 
chart (Figure 7) illustrates the entire process of Interactive problem 
solving using the knowledge matrix. He shall conclude this general 
discussion of the Interactive expert system by comparing the characteristics 
of a conventional expert system with the interactive model. As we stated 
previously, the conventional expert system Is essentially a branching or 
decision tree model of problem solving. The formal properties of such a 
model are well known: It has the characteristics of a context-free. 
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phrase-structure graiwiar. One of the most powerful characteristics of a 
context-free, phrase-structure granraar Is that It Is deterministic; i.e., 
there is only one possible solution for any given problem. 

On the other hand, the interactive expert system model Is a matrix 
rather than a tree. It is non-deterministic: It identifies a finite number 
of potential solutions (causes) to the problem and associated with each, a 
probability of the solution being the correct one. The identification of 
the correct solution is essentially made by trial and error, albeit an 
educated trial and error. The interactive model requires much greater 
contribution from the user than the classical model does. In the classical 
model, the user's task is to choose among mutually exclusive and exhaustive 
alternatives at each step in the diagnostic process. In the interactive 
model, the user must weigh a number of independent (but interactive) 
variables against each other in the process of making the best guess about 
what the problem Is. In this sense, the classical model of expert systems 
comprises a deductive process whereas the Interactive model comprises an 
inductive process. 

While the classical model is theoretically more powerful (since it 
guarantees a solution), in practice It reduces to, at best, a trial and 
error system because the model rests on the assumption that all choices 
within the decision tree are mutually exclusive, exhaustive, and decldable 
by the user. The interactive expert system, we believe, is a much more 
realistic model of hw experts actually make decisions under conditions of 
uncertainty. Moreover, it allows the user to make a number of trade-offs 
reflecting the conflicting demands and expediencies of real world 
situations. 

Interactive Expert System as a Job Performance Aid 

The nature of the knowledge matrix makes the interactive expert system 
a powerful job performance aid: 
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1. since each cell in the knowledge matrix Is free standing, the 
information in the cell can be updated or completely revised without the 
need to alter the rest of the matrix. Changes in procedures can be fed 
directly into the conputer program that controls the knowledge matrix, 
allowing for proi^t, reliable and relatively inexpensive updating. 

2. Each cell can be a window through which a variety of infonnation 
can be transmitted. For example, a cell for a given Test Set displays 
several types of tests for that problem. The test names could be used as 
menus, which, when selected by the user, give additional information about 
the test: how to conduct it, special equipment needed, time required, where 
to go to get further information about the test, etc. In a cell for a 
Treatment Set, the menu could Include information about other components to 
check or idjust as a result of the correction of the original fault — a 
critically important point since so many systems Interact with other 
systems. 

3. Unlike a classical expert system which sequences its Information 
flow, the knowledge matrix arrays its information in a single display so 
that the user has immediate access to just those pieces of information that 
are Immediately relevant. 

4. As mentioned before, there are a number of trade-offs to consider 
In the process of diagnosing, testing, and correcting problenw. In order to 
make the best trade-off, the user must have a display of the alternatives 
and their consequences, a display that is highly compatible with a matrix 
format but very difficult to represent in a branching diagram. The 
knowledge matrix allows the user to consider all the options at one time. 

In a classical expert system, the user is allowed to see trees one at a 
time, but never the forest. 

5. The program should have the capability of recording the results of 
each use by means of a simple record keeping system. This information could 
be periodically consolidated at a common site and then used to update the 
entire system based on the actual live experience of the users. 
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A Case Stu4y 

To complete oup discussion of this primary function of the integrated 
model working In the aiding mode, we present an hypothetical case that of 
the accidental firing of a flare from the yet to be produced ALE-47 
Countermeasures Dispenser Set (CMOS) aboard an F-16, 

For our presentation, let us assume that an accidental firing of a 
flare has occurred on the flight line at Bentwaters AB, England. No 
personnel were hurt, no property destroyed (except one flare), and the 
maintenance team begins an investigation and plans for repair of the system. 
(Flares firing while aircraft are on the ground prompts serious reaction — 
people on the flight line could be killed; f1re can destroy an aircraft and 
even adjacent aircraft. The team wants to isolate the problem, fix it, and 
report that It was fixed quickly.) 

After entering Into the computer the problem (ALE-47 FLARE FIRED. A/C 
ON GROUND), the tasked personnel are presented with range and environment 
data queries. These data queries are answered, putting the problem and the 
surrounding events immediately into the data management system. (This 
action corresponds to Step One presented In the previous section.) 

The CRT then lists the framework for the Initial Symptom Set (ISS) on 
the left side of the screen and arrays the symptom sets of record which 
include the ISS as shown In Figure 8. 

This display tells the operator that based on only the three symptoms 
listed by him/her in the ISS, the problem has never occurred. Thus, the 
operator goes back and checks the Weight -on -Wheels switch (WOW) and the 
Safety Pin. The WOW is jumpered and the Safety Pin is not inserted. 
Confirming that two additional symptoms exist (which violates SOP), the 
operator then reads them into the display. The display then shows that only 
two faults contained in the organization maintenance system have these 
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symptoms, and that one of them has a 95X probability of being correct* 
(Since the system Is limited by Its Inputs, It notes at the bottom that, 
based on normal frequencies and the number In these cells. It predicts the 
curren.. «nlues above mv be wrong as many as 4 times In 10.) Further, it 
notes what test procedures are used to determine the fault. 

Most test procedures will be done at the test bench, and the test 
procedure Is called In two stages, the first of which Is the gloss which 
Indicates what Special Test Equipment (STE), time, and other costs may be 
Involved In the test. This allows the operator to decide among different 
tests when necessary. In our example (see Figure 9), these decisions are 
not really an issue. 

The test with the .95 probability Is chosen, and the operator (who has 
some pressure to know the right answer, and who has time) performs the fault 
Isolation test, finds that the Shop Replaceable Unit (SRU) Programmer 
Circuit Card Assembly (CCA) Is faulty, replaces It with a new one, then 
retests the Line Replaceable Unit (LRU) and finds that It works. 

The AID MODE would have made available a menu of the tests, diagrams, 
and other references In the library throughout this process— perhaps by a 
window at the bottom of the screen. 

When the operator comes back to the terminal and calls In the case 
number (probably by maintenance number or by a personal number), the 
terminal brings up the Test Procedure gloss and begins Its data logging 
queries, illustrated In Figure 10. (We assume In this example that other 
operators may have logged onto the AID system while the maintenance person 
was conducting test procedure 0233.) For the largest number of maintenance 
and troubleshooting activities, this AID system will do well and will 
provide tremendous service for a large number of installations. Earlier In 
this paper however, we had a branch on the flow diagram which ended after 
symptoms and causes could not be found -or v«re exhausted, and that was 
called DETECTIVE. We discuss it next. 
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The Detective System 



Thus far we have described a model of an Interactive expert system 
which can assist the user in diagnosing faults In complex equipment. The 
heart of this system was a knowledge matrix which provided the user with the 
known causes associated with the syn^toras the user identified (ISS), along 
with additional symptoms and tests that the user could use to confirm or 
dlsconfirm each potential cause. Based on a probability table and other 
practical considerations, the user went through the 11st of possible causes, 
one by one, using the tests provided until the correct cause was Identified 
and the fault corrected. 

In this section, we deal with the situation in which the user has gone 

through the possible causes listed In the knowledge matrix and has 

eliminated all of them without solving the problem. In this situation, one 
of the following three conditions must be true: 

1. The solution to the problem is, in fact, in the existing knowledge 
matrix, but for some reason the user overlooked It. Me call this condition 
"user error." 

2. The solution to the problem is in the interactive expert system 
data base but not in the knowledge matrix that the user called up. We call 
this "symptom error," because the user did not use the right configuration 
of symptoms In the ISS. 

3. The solution to the problem does not exist in the Interactive 
expert system. That is, even 1f the user were to go through every cause in 
the system one by one, the user would still not find the solution to the 
problem. We call this "system error," 
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The DETECTIVE system Is a procedure for exploring each of these three 
error conditions 1n a manner that Is the most practical help to the user. 
The difficulty Is that although the user knows that the solution of the 
problem is hidden by either user error, symptom error, or systea error, 
she/he has no way of telling which one it Is. The DETECTIVE system rests on 
the assu[n)t1on that a systematic search through the three conditions is more 
productive than the other aUemative, unsystematic trial and error search. 
The DETECTIVE system Interacts with the user in quite different ways in each 
of the three conditions. Accordingly, we discuss each condition separately: 

1. User error . There are two places where a user error Is likely: 

(a) a mistake In man-machine interface, such as a format error in calling up 
the program or a data entry keypunching error or (b) a procedural error in 
performing a verification test. These errors may be identified by 
inspection and by repeating the procedures. Mundane as these errors are, 
everyone with programming experience knows how easy It is to overlook 
continually the most elementary mistake. 

2. Symptom error . There Is only one type of symptom error: A symptom 
wjs entered into the ISS that should not be there {I.e., we have a false 
eymptom). The reverse -- not entering a symptom that was. In fact, present 
but which was overlooked — Is not a source of error. To see why this 1s 
true, consider how the interactive system works. Suppose that In creating 
the ISS to describe a particular equipment fault, the user Identifies 
symptoms A, B and C, but overlooks symptom D. In generating the knowledge 
matrix, the system will call up those causes from Its data base that share 
the symptom set A, B, and C. Omitting symptom D does not eliminate any 
possible cause from the knowledge matrix. The only harm that has been done 
is that the user nwst go through possible causes which would have otherwise 
been eliminated from the knowledge matrix if the system had searched for 
causes that share four symptoms rather than Just the three. 
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Suppose, however, that s^inptom C in this same example Is a false 
symptom; It has nothing to do with the cause of the equipment failure. For 
example, suppose C was a false signal created by a malfunction In the system 
that monitors the performance of the equipment, When C was entered Into the 
ISS, the system automatically excluded fr<»!i the knowledge matrix any cause 
that did not have symptom C associated with it. Thus, a number of potential 
causes, including in this case the actual cause, were not entered into the 
knowledge matrix. 

The user can check for symptom error by removing the syn^itoms from the 
ISS. If all the symj^-oms were removed from the ISS, the system would call 
up every cause in its data bank. A more practical approach would be for the 
user to remove the symptoms fro the ISS one at a time beginning with the 
syiT^toffl in which the user has the least confidence. Fo. example, if the 
monitoring system in the example above had a history of false signals, the 
user might be inclined to check symptom C before the other symptoms. 

3. System error. Still, we must confront the problem that in some 
situations, and particularly with new systems, there will be problems no one 
has yet encountered. There is no experimental base; therefore no knowledge 
matrix has been established. We shall addr^-.ss this error after the 
following discussion. 

The Interrelationship of Epi steirol ogy and Heuristic 

Important in this discussion is our presupposition that an epistemology 
(theory, classification of knowledge) is indeed a heuristic (system of/for 
discovering). To demonstrate this by analysis would require more space than 
we have available here, but since the point is important, we digress to 
present briefly the argument behind such a demonstration. 

Of the three major questions askad by rhetoricians throughout tliro 
—Whether something is. What kind of th^ng it is. How can we know it (also 
known as the ontologlcal, axiological, and epistemological questions) — the 
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epistemologlcal has remained the most coflmonly asked. Indeed, for the 
purpose of training, the epistemologlcal Is crucial. That the crucial 
question can unite the data base and the heuristic probes In the Integrated 
system of aiding, training, and assessing Is one of the bonuses of creating 
such a system. 

The frequency drivers (Figures 11 and 12) are the basis for an 
epistemologlcal system. That Is, they allow one to code and classify 
information Into unique, retrievable categories. The categories are unique 
In that Inputs are coded such that ea**-h maintenance action on each piece of 
equipment Is Individually addressed. The categories are retrievable In that 
the specific address, when cross-referenced to all the res of the 
Information Input about the activity or the equipment, can be q'Hckly 
Identified and pulled from the systan, complete with the richness provided 
by nearly Instant access to all other related Information (that Is, related 
by being an activity on similar equipment, similar activity on dissimilar 
equipment, similar time between maintenanre activities, same system with 
slightly different activities but with the same symptom sets, and so on). 

This extraordinarily rich system becomes a heuristic by the use of 
generative grammatical principles and generative rhetorical principles. 
Generative grammatical principles, especially those of the so-called rewrite 
transformations, provide questioning strategies which turn the cell and/or 
the data into meaningful questions. For example, a cell labeled 
(electronically) "Range** has these questions ouilt into it: What 
maintenance activity (since last maintenance activity) is this? When was 
the fault noted? When was it repaired? By what action 'vas it repaired? By 
whom? Where? What a.'e the relationships anmng this cell and the other 90 
cells? (That is, what arithmetic extrapolations may be derived from these 
relationships in other equipments, other similar scenarios, etc?) Was this 
activity also conducted at the subsystem and system levels in this or other 
similarly configured aircraft? When? Where? And so on. 
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Generative rhetorical principles work In the coding and subsequent 
retrieval (on demand) of Information about the range, environment, 
distribution, con^leinentary variation, definition, causes and effects, and 
agencies which alloiwd the cause to have the effect. Put more simply, the 
entire matrix Is, In effect, a rhetorical probe. 

The base of the Frequency Matrix (Figure 12) contains elements of 
both the Burkean and the Kintschlan systems of rhetorical probes. The right 
sloe of the matrix incorporates the Young, Becker, and Pike probes, which 
are commonly known as the tagmemic heuristics or "tagmemic rhetoric," and 
which prompt the Interactions which give data on complementary variation, 
extent, and distribution. The upper left side Is a distillation of Kline's 
model, specifically that aspect of the model which Involves questions about 
any activity which moves systems Into or out of stasis. (For a more 
complete discussion of the matrix, see Kline and Huff, 1983.) The entire 
system is an eclectic heuristic. 

The coding system Is an epistemologlcally derived system; the model 
into which it Is cast constitutes a rhetorically based heuristic. In that 
way, the proposed integrated systsn of aiding, training, and assessment 
participates simultaneously In being an epistemology and a heuristic. And, 
It Is that Intense interaction of systemic features which provides the 
Intellectual force for the third level of DETECTIVE. 

After the DETECTIVE system has verified, through authentication and 
field enlarging, that the probability of Type 1 (User Error) and Type 2 
(Symptom) errors, respectively, has been eliminated or greatly reduced, a 
point is reached at which the focus of machinennan Interface will, once 
again, shift. While In aiding mode, the machine provided data and little 
prompting; while in user and symptom detective error mode, the machine 
interacted with the human to review the decision processes of the AID mode 
and to elaborate the field. Now the machine becomes a prompting tool and a 
data recording device. 
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At this point, the solution Is not In the machine, or — If It be there 
— It Is not retrievable due to an inadequate address. In either case, the 
phrase "system error" Is appropriate since It signals that the problem 
exists due to the Inadequacy of the system. 

After the CRT informs the operator that it is shifting Into the system 
error mode so that others may study the problem, 1t begins to show sets of 
heuristic questions which the operator is to answer. This third level 
routine is designed to poll the operator for all available knowledge about 
the fault or problem. 

The Integrated system's generative grammar operates on the system's 
Implied lexical sets to generate queries. The queries when answered are 
stored (both Q»A) and become part of the parameters used in fixing the 
probability estimates given in aiding, become new/expanded data for the 
system, and occasionally prompt the operator to reenter the AID or DETECTIVE 
sequences because the fresher picture of the problem produced refinements 
in the ISS or the fault sets. 

A single Interrogative transformation subroutine Is all that is needed. 
For example, if the problem has been Isolated to a single LRU, our matrix is 
narrowed to the multicellular probe, shown in Figure 13, which becomes 
the lexical sets which, by action of the interrogative subroutines, yield 
(as an example) the questions of Figure 14. Once these questions have been 
answered the system will ask the operator to update user error, update 
symptom error, and return, if appropriate, to the ISS to rerun the program 
with the new information included. 

If the operator does not return, the machine assigns a unique number 
to the case. This unique number is read into the level four training menu 
(after the case is reviewed by the maintenance chief); this unique number 
also marks the case with a direct system address, allowing the case to be 
found quickly. 
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FIGURE 13 PROBE NARROWED TO SINGLE LEVEL OF CELLS 
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CAUSE 
(ACTOR) 


TOOL 
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WHAT UN(T(S) CAUSES / 

THIS UNIT TO OPERATE? / 

ARE ANY OF THE CAUSES / 
NOW UNDER REPAIR? / y 

LIST POSSIBLE CAUSES X X 

OF THIS FAULT . ^/ / 

WHICH IS MOST LIKELY? ^ ^ 



WHAT MAINTENANCE 

ACTIVITY HAS OCCURRED 

ON THIS L/SRU SINCE 

LAST SYSTEM MA? 
WHAT DOES THIS L/SRU 

CAUSE? 

L/SRU CONTINUES 

TO MALFUNCTION . WHAT 

WIU HAPPEN TO 

THE SYSTEM? 






WHAT CAUSES THIS 

L/SRU TO OPERATE 

UNDER NORMAL 

CONDITIONS? 
WHICH OF THESE 

CAUSES ARE 

PRESENT AT 

THIS TME7 
WHICH ARE NOT 

PRESENT? 
WHAT DOES THIS 

UNIT CAUSE TO 

OPERATE m THE 

NEXT HIGHER 

SYSTEM? 
IS THAT UNIT 

WORKING WELL? 



\ HAVE ALL QUERIES 

\ BEEN ANSWERED? 

\ ANY ADDITKMAL 
v V INFORMATION YOU CAN 
X \ GIVE MWHT BE HELPFUL. 



HAVE ALL QUERIES 
BEEN ANSWERED? 
ARE ANY RANGE-ORIENTED 
ASPECTS MORE SUSPECT? 

WHICH? 

ANY OTHER INFORMATION 
ABOUT THIS L/SRU 
MIGHT BE HELPFUL. 
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INTERROGATIVE ROUTINE PROVIDES, PROMPTS 
AND QUERIES IN THE DETECTIVE MODE. 

FIGURE 14 INTERROGATIVE ROUTINE PROVIDES PROMPTS 
AND QUERIES IN THE DETECTION MODE. 
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Finally, the operator Is Instructed to log off. 

Whenever a solution Is found, the case Is written up corapletely and 
sent back through the system to the address. In this way, the newly 
discovered solution Is quickly Integrated. The unique case nurnber Is erased 
when the solution is encoded; this takes the case out of the level four 
training menu and leaves the case In the data hank. 

In these ways, job aiding with continuously updating data bases 1s 
possible. It Is these updated data bases which provide the rich source of 
training materials, the features of which are shown in Figure 15. 

B. Training 

Training is crucial to any organization, and certainly the Air Force is 
no exception. The training program nust provide realistic trainii ], thus 
eliminating the "Transfer of Learning" problem. It must provide up-to-date 
instruction which uses all current Technical Orders (TOs) and Training/ 
Technical Manuals; thus it must be rapidly and directly updatable. It must 
provide trainees with opportunities for additional, personally chosen 
training; thus, it must be both user-friendly and accessible. Our 
conception of the Integrated Aiding, Training, and ^^"pssing Model provides 
these features. 

The training system is drawn from the military training model. 
Instructional development is a twelve step process in our view. 

1. Perform task/job analysis. 

2. Draft preliminary objectives (Performance objectives; who does 
what, with what, to what level?). 

3. Set and confirm prerequisites for instruction. 

4. Determine students' background (in general). 

5. Revise objectives. 

6. Design test procedures. 

7. Select materials. 
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TRAINING 



, REALISTIC, HANDS-ON 
. UP-TO-DATE 
. USER-FRIENDLY 
• ACCESSIBLE 



FIGURE 15 FOUR FEATURES OF TRAINING 



8. Sequence materials. 

9. Select Instructional niethod(s). 

10. Teach. 

11. Test. 

12. Revise as necessary to achieve progpam goals (100/100%, 90/100%*). 

Instructional development, additionally, must fit into the Integrated 
Logistics Support (ILS) system. Within the ILS Model, there are provisions 
for three major influences/inputs into training: Instructor, ILS data 
(which include Technical and Training Manuals and Reliability and 
Maintainability Ojectives), and training course design (usually provided by 
contractors). Thus, the Air Force and the contractor cooperate on the first 
five steps; the contractor provides the next three; and the Air Force the 
four following. The integrated model is focused on the AF delivery portion 
(the last four steps). 

The instructional methods we incorporate are direct teaching (via 
manuals ind other printed materials and Instructor presentation), case 
study, and problem solving. We shall omit discussion of the Instructor in 
the balance of this paper since the system is designed to provide meaningful 
On-the-Oob Training (OJT), which supports an instructor if one is available. 

How the Training Subsystem Works 

When a trainee keys into the CRT ~ whether done because an instructor 
or superior required the work or because the trainee wants to know about a 
unit or system upon which work may someday be required — he or she is 
presented a menu which lists the available materials about the systems and 
subsystems. After keying to the (sub) system, the trainee finds a second 
menu; this one has an option for training and one for job aiding (plus 
others which may later be added). The training option is selected. 



*10D% of trainees score 100% correct, 90% of trainees score 100% correct. 
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The CRT then displays a message something like this 



Al/C SMITH: 

FOR TRAINING YOU HAVE THE OPTION OF GOING DIRECTLY TO THE CASE STUDIES 

AND ANALYSES OF MAINTENANCE ON THE LRU/SYSTEM OR USING THE 

LIBRARY TO READ AND STUDY ABOUT LRU/SYSTEM. DO YOU WISH TO USE THE 
LIBRARY? Y/N 

The library has indexed options for all the "Ws, TOs, Ilustrated Parts 
Breakdowns, charts, graphs, and schematics — along i#1th recoamended reading 
lists for introductory, advanced, and curren- awareness reading. The 
library also has a sample listing of job aids available on the indexed 
subject. The trainee may read the materials on the CRT or by securing a 
hard copy (on a remote duty station); for current awareness materials, the 
system may be instructed to print any material. After library use or 
instead of using It, the trainee can move on to the case study problems. 

The case studies are drawn from the Job aids. In the training mode, 
both the solution and the algorithm are being taught. Each of the steps in 
the job aid Is available in turn — when the trainee requests them. These 
options can be presented after each step is con^leted. 

In the training mode case studies are arranged by level of difficulty 
(see Figure 16) into four sets: 

Introductory. In these, the solution is the option which has high 
probability and cost-effectiveness. 

Intermediate. In these, the solution is discovered to be a lower 
probability (with or without cost trade-off). 

Advanced. In these, the solution Is found during the heuristic search, 
and it is of mid to high probability. 
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FIGURE 16 CASE STUDY ARRANGEMENT 
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Yet Unsolved. An acfal, unsolved problem currently under study. 



While solving a problem In a case study the trainee will have all 
standard job aid tools available (and, perhaps, an instructor). 

Trainees can produce a hard copy of their completed case study (to 
present to their Instructor or the maintenance chief) or of the case per se. 
For level four cases, the Air Force may vflsh to offer a cash prize for a 
defensible solution; this *<ould be to encourage discussion of the problem — 
especially discussion by veteran flight-! Ine non-commissioned officers 
(NCOS). 

In addition to the case studies, which are a form of testing, the 
trainee can call up self-tests from a menu. These are drawn from training 
manuals, SOPs, and the case studies and are the tests which may be used In 
resident Instruction classes. 

They serve as performance indicators for "passing out" of a training 
requirement; they also allow continuing availability of an array of tests on 
basic avionics circuitry, logic and microwave circuit fundamentals, 
electronic warfare/command-control -communication Interfaces, fuels, and 
other Issues of concern. These tests can be used by NCOs and officers as 
screening tools for replacements, by anyone for self-testing (before 
beginning a course, for example), and by Instructors as a source of dally 
tests and pre-/post-tests. 

When case studies and supplementary tests are packaged separately, they 
become the battery for performance assessment. 

C. Performance Assessment 

Interrelationship of Teaching and Testing 

An especially Important presupposition In the model Is the direct 
interrelation of teaching and testing: An> procedure, protocol, or exercise 
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which can be used as a teaching device can also be used as a testing 
tool. 



For example: Teach 2 + 2 » 4; 2 + 3 « 5. 

Test 2+2-?,;?.+ 3«5. 

Or: TEACH The characteristic Inpedance of a parallel transmission 

line may be determined by solving ■ 276 log b., 

a 

where b Is the distance between conductors and a is the 
diameter of one of the conductors. 

TEST What Is the characteristic impedance of a parallel 

transmission line with four Inch spacers and .023 drawn 
copper wire conductors? 



This presupposition Is important because the elements of the training 
exercises become part of the battery offered for performance assessment. 
The machine has no problem with this translation, since the conversion rules 
are those of transformatio.til grammar, notably the "T-WH," "T-DO," and 
"T-BE, Present" rewrite rules. 

Performance assessment involves many considerations, ranging from 
aptitude, general knowledge, to time in service/grade/ role, in order to 
actually demonstrate performance. The case studies and the array of tests 
available in the training mode, added to other, already existing performance 
assessment tools provide an enriched and constantly updatable source of 
tools for assessing performance — both actually demonstrated and potential 
performance. 

For routine screening, the introductory case studies from training 
would be more appropriate as frames for determining whether the 
troubleshooting algorithm has been Internalized. The actual training case 
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stu4y may be used to authenticate perforroance capabilities. And advanced or 
yet unsolved cases may be used as exit criteria for noting whether higher 
level (training or AFSC) assignment Is warranted. 

We believe it is important for individuals to have access to 
performance assessment batteries, i.e., self -assessment. This is an 
especially Important consideration when it can be coud&ined with a 
self -directed training program. In this integrated model that is possible. 

Richardson (1983) pointed out: 

The challenges are to move performance measurement into the operational 
environments and to integrate it with personnel iranagement and training 
organizations' programs. The development of symbolic substitutes, 
based on modern maintenance simulation work (seen as Interactive 
graphics simulation utilizing Intelligent video discs), also provides a 
challenge, (p. 35) 

The proposed integrated model, in fact, does move this role into the 
operational environment. 

Special attention is due for Richardson's second "challenge." The 
routine use of the (anticipated) software-based simulations of virtually all 
test, maintenance and troubleshooting systems /procedures will be focused 
particularly on performance assessir^nt roles. With the updatable. 
Integrated system, we anticipate that the use of simulations will be 
optimized. 
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IIU UPDATING THE SYSTEM 



Although it is somewhat beyond the scope of this paper to address the 
Infrastructure of the tool used to update this Integrated system, it Is 
nonetheless Important to dwell briefly on this problem. 

Ue provide conceptually designed frameworks for updating IRU/SRU fix 
probability (see Figure a1), for Mean Time Between Failures (MTBF) and case 
histories {see Figure 12), for Built-in Test (BIT fault Isolation tracing 
(see Figure 17), and for system scanning data (see Figure 18). The lowest 
level Is the frequency count which drives the fix probability table (using 
simple r statistics). Each datum has an unique address; so from the 
hlstogramnic data in the fix table an individualized SRU and LRU can be 
traced, even while maintenance data are being aggregated. The Fault 
Isolation Tree is driven by BITE, by system/subsystem flow diagrams, by TOs. 
and by user provided Information. Organizational support data are provided 
within the matrix. From the matrices of many maintenance activities can be 
drawn aircraft system, LRU, and SRU data, similar cause/similar 
effect/similar context data, as well as multiple test and repair data. 

Finally, system scanning Beta (the system studying Itself, that is) 
ensures routine Macro-MTBF data collection. 

Updating can be by direct input, by teletype, by batch load — all that 
is required is state-of-the-art software and prearranged availability (for 
example, monthly). 



46 




CMOS 
F-18 


0 




0 




SEQ SW 
CMOS 


0 




0 




03 SEQ 
SW 


R&R 










0 


112 






X 


1 
















60«»F 





150784 
1756R 



[i180784 
^1425R 



(LEGEND : SRU 03 OF THE SEQUENCER SWITCH . COUNTERI^SURES 
DISPENSING SYSTEM (ON AN F-16}. WAS REMOVED AND 
REPLACED. THE MALFUNCTK>N WAS NOTED ON JULY 15, 
1M4 AT 5:56 PM ; AT THE TftdE OF THE ACCIDENTAL FIRING, 
THE F-16 WAS ON THE GROUND (0 ALTTTUI^ STABLE. 
STATIONARY ATTITUI^) AND THE AMBIENT TEMPERATURE 
WAS eW. THE LAST MAINTENANCE ACTIVITY Cm THIS LRU 
WAS 112 FLIGHT HOURS AGO; THERE HAVE BEEN SIX 
MAINTENANCE ACTtVITSS ON THIS LRU (THE X IN SQUARE 6 
OF THE 'RANGE* BLOCK). THE SYSTEM WAS REPAIRED AND 
INSPECTED AT 2:25 PM ON JULY 16,1984). 



FIGURE 17 MATRIX TRACE OF SAMPLE SRU R&R 
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BETA = NUMBER X NATURE 




1 2 3 4 5 e 7 8 9 10 11 12 13 14 16 16 17 18 18 20 
FIGURE 18 MAINTENANCE ON LRU (BY ID NO.) 



IV. ANTICIPATING CLEARLY THE FUTURE 

The program envisioned in this paper Is not a solution to a problem 
which exists nov# and once solved will go away. Indeed, the problem win 
continue and win change. The solution, then, must be one which can change 
and can stin provide responses in meaningful formats. While are 
uncertain of the future and problems yet to become known, we have considered 
eight specific innovations /new problems: Four will be confronted we believe 
before 1995, and four will begin to be known and be confronted around and 
after 1995, 

A. Present - 1995 

1. "Smarter" ATE/STE. Systems alrea<<y under development and due for 
production in late 1988 and 1989 will have interactive software. This is 
especially true for specialized test equipment (STE) and somewhat true for 
automatic test equipment (ATE). For example, a design proposed for the 
ALE-47 countermeasures dispensing set will require only one piece of STE 
(and that is of a conplexlty currently within reach); it will require ATE 
currently in production with a new software package — this software will 
enable more testing, quicker fault Isolation, and corapatibnity with the 
ALE-47 software, which is extraordinarily complex. 

The E/EF/F-111 aircraft, for another example, will use In its EW suite 
some six major busses (not counting edge-light and main source busses). ATE 
used for fault Isolation will "read" all six busses simultaneously 
(necessarily) and Interactively. STE must, of course, do likewise. 

Smarter ATE and STE will draw heavily on AI breakthroughs currently 
being reported and tested. Systematic search routines, quicker because they 
are not serial or linear, are already being Incorporated. 
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The proposed system v*m work well with the new ATE/STE and will ha>flB 
compatible data Inputs/outputs (I/Os). The compatibility Is assured becjuse 
the matrix allows Interactive collection of Information. Only the addre'l^ 
must be coded. 



2. 'Natural language" I/O. Between the present and 199^, >eipred.1<^t 
that the current research in natural language I/Os will produj|e W word j 
Identification systems more nearly representative of a true.r^n^fi'al* ^ 
language system. The principal benefn of this Is user-frldtdlt|Jss. i 
(User-friendliness Increases as the nature and the quantity if ^ryptlor^ 
required decrease.) {. • » 



t 



While these systems and their related syntactic parseqs w® mo\f* 
much closer to natural language I/Os. actually achieving t|at l^te^B^|ns 
remote — at least within the next ten years. r v m ' ^ ' 

3. Expanding Data Base. The problem most difficult to jfnjigliie^m the 
data base in 1993 to 1995 (and b«yond). Current Air Force pros^a^ Will: 
still be Invaluable In that data base, but the newer systems j^ll- outnumber 
the existing ones. These new systems Include the small ICBM,^Jthe; revised 
MX. the B-IB, Stealth, the U-4, Integrated EW/RWR/VIS and MAD'jMI^'slle ^ 
Approach Detector Systems), PenAlds, the "new" UYK-19, Advanced . , 
MIL-STO-1750A CPU architecture — all these and more! ' / 

/ ' \ 

■ ■ r : 

Providing for this data base Is mind-boggling in terms of guantjty ^ut 
not concept. The proposed system will not be burdened by .be nit <J^a I 
systems. The major problem will be formatting Inputs — an^l this should'be 
levied against system contractors on the Contract Data Requirements Listi 
(CDRLs). Updating will then, of course, be required. * 



From 1995 to 2005 the system load will again shift as;thBj 
Anti -Sate! lite Weapons systems are completely replaced; this cf|»lacement. 



* • 

however, will not disrupt the proposed sysVem. ■< [, 
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4. Job Aiding, Training, and Simulators^ No doubt, simulators will 
become the most commonplace delivery vehicle for all forms of training. The 
single formula of available use + cost + learning dictates this clearly. 

Since this system Is designed to be based on job aiding and training, 
as newer simulators and simulations arise there will be no problems in 
adapting to them. 

5. Integrated Expert Systems and Local Area Networks {LANs). As the 
WMMCCS Information System (WIS) and the UUNA, AMIS, and NAVLAN (for, 
respectively, the Air Force, the Amy, and the Navy) come on line In 
1986-1992, the system proposed In this document will become fully 

deploy able. The distributed Intelligence characteristics of the WIS Blocks 
D through N suits the proposed system well. 

The Interface <tevice to be used in the Air Force architecture, and 
likely for the WWMCCS Information System Network, will probably not be 
application -specif 1c. Hopefully its procurement (probably amounting to some 
50,000 units In the years up to 1992 and the same number in the years from 
1992 to 2000) will be based on its versatility. The $.5B (or so) spent in 
the next seven years on those interface devices will, in fact, be the 
Implementing funds for a system such as the one proposed in this document. 

6. Test Bed. In order to prepare for the applications envisioned and 
to be ready for Blocks C and D of the WWMCCS Information System, where the 
real testing of both the Information System and the proposed model will be 
possible, we strongly recommend that the Air Force and AFHRL continue to 
support such programs and move them to test-bed phases. 

B. Beyond 1995 

In addition to the developments noted above, we anticipate at least 
four others to become critical In the decade or so beginning around 1995. 



U Qps BITE Download. Currently planned BITE Improvements Include 
ffloving data with BITE and "BITE-interfaced off-loading" from aircraft system 
to the test bench. Long a favorite Idea of dreamers and science fiction 
writers. BITE download Is now within reach. In this plan, the pilot, 
engineering officer, and/or the crew chief will transmit on-board BITE to 
the test bench, as they depart the A/C. {Not only A/C need be considered; 
any system can be Involved.) Test bench sets can poll BITE while the system 
Is In use. (One Interesting concept Is the continuously down-llnked 
telemetric BITE — stored for extraordinarily high-speed burst transmissions 
over encrypted carriers In tactical situations.) 

The system proposed In this paper Is compatible for the same reason It 
is In A-3, above. Formatting I/Os will be very time consuming and 
expensive. These will be levied on contractors as part of RSM Program Plans 
and of Software Specifications. 

2. Universal ICAI. "Universal" or, perhaps, "nearly universal" 
Intelligent Coraputer-Asr isted Instruction (ICAI) will be the case In the 
years beyond a995. ThiS use of Intelligent systems Is not a problem for the 
plan advocated In this paper. Rather, It Is the base towaru which this plan 
moves. 

3. Hemispheric Hook-ups. By 2005, Individual test benches probably 
will be part of hemispheric or worldwide networks. The Increasingly short 
turnaround pooling will be an asset to the Integrated aiding, training, 
assessing model. 

Since electronic intercommunication dees not present any technically 
Insurmountable problens now, we anticipate little problem In implementing 
this "Intercontinental test bench." 
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4. Hand-Held/Helmet^Mounted Units. These are not real technical 
challenges; they are "innovative" In the sense that they allow (and probably 
promote) remote accessing of flight-line gear. Problems in projection at 
varying light conditions are the problem presently and will continue to be. 
No problems are presented for the proposed system, however. 
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V. CONCLUSION 



The programming proposed In this paper requires further stu4y, 
especially in the areas of life <ycle costs, initial funding and 
dissemination, and software development. The model itself should be 
studied from four perspectives: 

- Routines and subroutines required 

- Matrix - address algorithm 

- Scope of library requirements 

- Plan for further study 



While budgeting and forecasting using current fiscal year dollars is 
virtually Impossible, we assert that. Including initialization, the Air 
Force (alone) can expect that savings directly attributable to this program 
will exceed $100 mi 11 ion per annum. Adding the Amy and Nayy to the Air 
Force in a Tri -Service Initiative would create extraordinary savings. 
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